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INTRODUCTION 



This paper provides a summary of a comprehensive evaluation of the Kentucky statewide 
Extended School Services (ESS) program. For complete reports of this evaluation, see Cowley 
and Meehan (2001) and Cowley et al. (2002). 



Background 

The Extended School Services (ESS) program was established in 1990 as part of the 
Kentucky Education Reform Act (KERA). Designed specifically to address the needs of 
Kentucky's at-risk student population, ESS is an aggressive, proactive program for addressing 
academic problems before they become ingrained (Nesselrodt & Schaffer, 2000b). The ESS 
program extends the school day, week, or year for students at risk of academic failure, providing 
them with additional instructional time to help them meet academic goals. Rather than being an 
“add-on” or “stand-alone” program, ESS is designed to be an integral part of each school's 
regular academic program, thus ensuring that students receive instructional assistance in core 
content subjects in which they are perfonning poorly. 

The major emphases of the statewide ESS program are to (1) sustain students’ present 
level of performance to prevent them from falling behind; (2) provide extended programming for 
students who have been retained or are at risk of (a) being retained in a class or grade or 
(b) failing to graduate on time without assistance; and (3) close the achievement gap of low- 
performing students so they will perform successfully in the program appropriate to their age. 
According to publications from the Division of Extended Learning of the Kentucky Department 
of Education (KDE), nearly every school provides such services; thus, about 1,450 schools have 
some type of ESS program (AEL, 2001; Quality Education Data, 1998). 



Current Evaluation 

In the fall of 2001, KDE contracted with a partnership of the Appalachia Educational 
Laboratory (AEL) and Western Kentucky University (WKU) for a comprehensive evaluation of 
the ESS program during the 2001-02 school year. Fifteen evaluation questions were assembled 
into five major topics: (1) identification, referral, and assignment of services; (2) profiles of 
students receiving services; (3) profiles of ESS programs and their implementation patterns; (4) 
services to students placed at risk; and (5) ESS implementation patterns and outcomes. This 
evaluation design was based in part on a previous pilot-test evaluation conducted by Nesselrodt 
and Schaffer (2001a, 2001b). 

AEL's evaluation of the ESS program utilized two major components — statewide surveys 
and site visits. These components were broken down into five main phases: statewide surveys, 
training session for site visits, falEwinter site visits, summer visits, and data analyses. The 
surveys were administered to the district and school ESS coordinators in the fall of 2001. See 
Figure 1 for a graphic portrayal of AEL’s evaluation of the ESS program. 
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Summer Site Visits 

• 6 of 18 Sites Above 

• Same Surveys, Interviews, 

Observations, and Forms less the 
District ESS Coordinator Interview 
and the Non-ESS Teacher Survey 



Figure 1 : Graphic Portrayal of the Evaluation 
of the Kentucky Extended School Services Program 
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The major purposes for the site school visits in the fall/winter of 2001-02 and the summer 
of 2002 were to gather intensive, extensive ESS program data that would be generalizable, valid, 
and reliable to Kentucky programs statewide and to corroborate findings from the statewide 
surveys previously administered to district and school ESS coordinators. The site visits 
replicated most of the procedures and data collection instruments utilized in the pilot-test 
evaluation. Pairs of trained data collectors made two to three day visits to a sample of 24 schools 
with ESS programs (18 during the fall/winter of 2001-02 and 6 in the summer of 2002) to collect 
both qualitative and quantitative data from a variety of ESS stakeholder groups.* See Table 1 
for descriptive information about the sites selected for visits. 

The site visit data collection involved classroom and ESS session observations; 
interviews with ESS teachers, ESS students, ESS parents, the school ESS coordinator, and the 
ESS district coordinator; surveys of non-ESS teachers, ESS teachers, ESS students, and ESS 
parents; a school and program description fonn; and written documentation such as the school's 
consolidated plan and needs assessment, as well as descriptions/policies of the ESS program. 
AEL added two new instruments: the Innovation Component Configuration Map, to generate 
patterns of implementation across ESS programs, and the AEL Continuous School Improvement 
Questionnaire, to measure the extent to which a school faculty is committed to continuous 
improvement. 



*A two-stage sampling process was implemented to identify the 24 schools. In the first stage, 
KDE staff established a pool of 48 schools through a six-step process that reviewed student 
achievement data, percentage of students eligible for free or reduced-price meals, overall academic 
student index, ethnicity, school-level perfonnance indicators such as novice-level readers and 
dropout rates, comparisons of subsets of student scores within schools, and geographic and 
demographic representations. AEL completed the second stage by securing Johnson locale codes 
(National Center for Education Statistics, 2001) and published enrollment figures (Quality 
Education Data, 1998) for each of the 48 schools. Using a combination of building level, 
geography, urbanicity, and enrollment, AEL staff selected the 18 schools for the fall/winter 2001- 
02 visits. AEL staff collaborated by telephone with KDE staff to identify which 6 of the 18 
schools would be revisited during the summer of 2002, based on geography, building level, and 
general representativeness of Kentucky ESS programs. 




Table 1: Descriptive Information on School Sites Selected for Visits 



School 

Number 


Achievement 

Gap 


Visit Time 


Grades a 


Rounded 
Enrollment a 


Attendance 
Rate a 


F ree/Reduc ed-Price 
Lunch (Title I) b 


Student to 
Computer Ratio a 


Locale Type c 


1 


Minimum 


Fall and Summer 


9-12 


1,330 


94.7% 


17% 


5.2:1 


Mid-size City 


2 


Large 


Fall and Summer 


9-12 


960 


95.1% 


31% 


3.5:1 


Large Town 


3 


Large 


Fall 


K-5 


470 


95.5% 


42% 


4.7:1 


Large City 


4 


Minimum 


Fall 


9-12 


1,550 


96.0% 


14% 


4.6:1 


Urban Fringe of Large City 


5 


Minimum 


Fall and Summer 


PK-6 


270 


94.1% 


72% 


6.0:1 


Rural, outside MSA 


6 


Large 


Fall 


3-8 


370 


94.6% 


39% 


4.0:1 


Small Town 


7 


Minimum 


Fall 


K-6 1 


300 


96.2% 


27% 


3.3:1 


Rural, outside MSA 


8 


Large 


Fall 


K-5 1 


640 


96.0% 


31% 


3.9:1 


Large Town 


9 


Large 


Fall 


9-12 


1,550 


92.8% 


15% 


3.0:1 


Mid-size City 


10 


Minimum 


Fall and Summer 


9- 12 1 


950 


96.4% 


54% 


5.0:1 


Small Town 


11 


Large 


Fall 


9- 12 1 


1,760 


89.0% 


16% 


5.0:1 


Mid-size City 


12 


Minimum 


Fall 


6-8 


420 


94.6% 


50% 


3.9:1 


Small Town 


13 


Large 


Fall 


6-8 


700 


94.2% 


16% 


5.0:1 


Urban Fringe of Large City 


14 


Minimum 


Fall 


K-5 1 


390 


95.7% 


42% 


5.1:1 


Small Town 


15 


Minimum 


Fall and Summer 


OS 

i 

OO 


700 


94.5% 


18% 


4.2:1 


Rural, inside MSA 


16 


Minimum 


Fall 


7-8 


450 


94.1% 


38% 


5.1:1 


Rural, outside MSA 


17 


Large 


Fall and Summer 


K-5 1 


540 


95.5% 


47% 


4.0:1 


Small Town 


18 


Large 


Fall 


OS 

I 

OO 


750 


95.7% 


15% 


6.0:1 


Urban Fringe of Large City 



a From “School Report Cards,” by Kentucky Department of Education, 2002, Retrieved June 2, 2003, from Kentucky Department of Education website: 
http://aaps.kde.state.ky.us/report card/ 

b& 1 From QED 2001-2002 State School Guides: Kentucky (19 th ed.J, by Quality Education Data, 2001, Denver, CO: Quality Education Data. Copyright 2001 by Quality 
Education Data. 

c From Public Elementary/Secondary School Universe Data in Common Core of Data, by National Center for Education Statistics, 2002, Washington DC: Author. 
Available from http://www.nces.ed.gov/ccd/pubschuniv.asp 
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METHODS 



The first task for this comprehensive evaluation was to convert all of the instruments used 
in the Nesselrodt and Schaffer pilot test (2000a, 2000b) to a machine-scannable format 
(excluding the interview protocols) and to make improvements based on lessons learned from the 
pilot test and suggestions from KDE staff. These instruments included six surveys (district 
coordinator, school coordinator, ESS teacher, non-ESS teacher, ESS student, and parent of ESS 
student); five interview protocols (district coordinator, school coordinator, ESS teacher, ESS 
student, and parent of ESS student); the Special Strategies Observation System (SSOS), which 
included three fonns related to classroom observation, quality of instruction, and environmental 
resources of the classroom; and a school and program description form. AEL staff also selected 
and prepared in scannable format two other data collection instruments: the AEL Continuous 
School Improvement Questionnaire (AEL CSIQ) and the Innovation Component Configuration 
Map (ICCM). This first task was completed by August 2001 and all instruments were submitted 
to the AEL Institutional Review Board (IRB) for approval. A few minor changes were made in 
response to IRB suggestions and all instruments were approved for use in the evaluation. 

The first major component of the evaluation consisted of statewide surveys to all 179 
district and 1,433 school ESS coordinators in the fall of 2001. For a complete report on the 
instrumentation, data collection and analysis, and findings for the statewide coordinator surveys, 
see Perceptions of Kentucky’s Extended School Services Program by District and School 
Coordinators (Cowley & Meehan, 2001). 



Training Session for Site Visits 



An experienced data collector was identified to conduct the training session; hired in 
September, the consultant was instrumental in helping revise the SSOS instruments. Next steps 
included designing the training session, developing the training manual, and hiring the data collectors 
(six AEL staff, four Kentucky educators/consultants, and two West Virginia consultants). 

A training manual was developed for data collectors to use during and after the training 
session. This manual contained copies of each instrument to be used in the evaluation, along 
with instructions for administration. It also included an agenda, a participant list, an overview of 
the ESS project and evaluation, sections for each of the major types of data collection, 
procedural information (random selection information, student consent forms, site visit 
procedures, materials checklist), a calendar for scheduling site visits, and an evaluation form for 
the training session. An e-mail listserv was developed so that data collectors could interact 
quickly and easily with one another and with AEL evaluation staff during the evaluation project. 

The training session took place on October 3-5, 2001. Twelve individuals completed the 
three-day training. The bulk of the training time was devoted to the three instruments in the 
SSOS. Discussion of codes, practice with classroom videotapes, and instructions proceeded on 
schedule. One important aspect of the training was the inclusion of actual practice observations 
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conducted in classrooms at a nearby high school. After each live classroom observation during 
the second and third days of training, a debriefing session was conducted to discuss coding 
questions and concerns. The training also dealt with other instruments such as interviews and 
surveys. The AEL director of evaluation conducted the sessions on the ICCM and AEL CSIQ. 

Several wrap-up activities were completed at the conclusion of the training session. One 
was an evaluative activity in which participants coded a criterion tape of classroom behavior 
previously selected by the trainer, who reviewed each completed SSOS and verified that all 
participants met or exceeded the established level of competency in terms of coding 
specifications. A second wrap-up activity was making tentative pairings of data collectors and 
schools, based on interests, schedules, and geographical proximity. The third and final wrap-up 
activity was the completion of an evaluation fonn by all participants. 



AEL CSIQ Administration 



Instrumentation. The AEL Continuous School Improvement Questionnaire (AEL 
CSIQ) is a 60-item instrument that measures the extent to which a school’s faculty members are 
committed to continuous improvement. The 60 items are each rated on a Likert-type scale of 
1 to 6 (not present to present to a high degree ). These items comprise six scales: (1) learning 
culture, (2) school/family/community connections, (3) shared leadership, (4) shared goals for 
learning, (5) purposeful student assessment, and (6) effective teaching. This instrument was used 
to discern the connection between faculty members’ commitment to continuous improvement and 
implementation of their ESS programs. 

Data collection. During the second week of October, AEL staff drafted a letter for the 
Kentucky Commissioner of Education’s signature, soliciting cooperation from the 48 schools in 
this step of the evaluation. AEL staff then assembled kits containing the cover letter, the 
instruments, and a postage-paid return envelope; these kits were mailed on October 15. AEL 
staff called all non-responding schools and spoke with the principal or school coordinator to 
solicit infonnation regarding the status of the instrument completion. In late January, KDE staff 
contacted the few remaining schools. By the end of February, 47 of the 48 schools had returned 
completed forms, for a return rate of 98%. 

Data analysis. AEL staff scanned the surveys using Remark optical scanning software, 
cleaned the data files, and exported them to SPSS for statistical analyses. Analysis was 
conducted at the scale level, and appropriate descriptive statistics were generated. Analyses 
were generated for each individual school and for aggregated building-level profiles for 
elementary, middle, and high schools. Data were also analyzed to determine whether significant 
differences existed by ESS implementation patterns. Descriptive statistics and one-way Analysis 
of Variance (ANOVAs) were utilized at the scale level for this analysis.* 



*Random sampling was not utilized in this evaluation, given the original sample pool of 48 schools provided by KDE. Therefore the assumption 
for the use of ANOVAs and t tests does not apply in a classical sense. However, they are informative for comparing within-group variance to 
between-group variance for some of the measures used in this evaluation. 
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Validity and reliability. The validity and stability reliability of this instrument was 
established in prior research (Meehan, Cowley, Craig, Balow, & Childers, 2002). As a measure 
of the internal consistency reliability of the instrument for this administration, Cronbach alpha 
values were computed for the 10 items within each of the six scales, as well as for all of the 
items in the instrument. For this set of scores, alpha values ranged from .89 for the learning 
culture scale to .94 for effective teaching; the overall value for all items was .98. 



Site Visits 



To begin the site visit process, AEL staff drafted a letter for the Commissioner’s signature, 
announcing to the 18 schools (6 elementary, 6 middle, and 6 high schools) that they had been selected. 
KDE staff mailed the letters in October 2001, and data collectors began contacting schools to schedule 
their visits. By the second week of November, all of the visits were scheduled for completion by the 
end of February 2002; most data collection teams were scheduled for three site visits. Of the original 
schools selected for the falFwinter site visits, one was eventually replaced due to a scheduling conflict. 

On April 25, a conference call involving AEL and KDE staff was held to select the 
summer site visit schools and two alternates. The decision was made to revisit 6 of the original 
18 schools (2 elementary, 1 middle, and 3 high schools). Such a strategy would provide data 
from the schools’ regular after-school ESS programs and their summer programs. KDE staff 
contacted an administrator at each school and secured their participation. 

All six visits were conducted during June 2002. In terms of methodology, the only 
differences in the summer site visits were the exclusion of the non-ESS teacher survey, the 
decision not to re-interview the district administrator, an increase in the targeted number of 
parent and student interviews, and an increase in the number of ESS students observed. As well, 
a few of the interview questions for parents, students, ESS teachers, and school coordinators 
were eliminated because they were not relevant to the summer programs. 

At the conclusion of each site visit, data collectors returned materials to AEL. Each set 
of school materials was logged in and checked for completeness. Data collectors were notified 
of any missing materials; completed materials were filed and stored by school. All signed 
consent fonns were sent to AEL’s Administrative Services Office for archival purposes. 
Interview notes were submitted to AEL support staff for typing before qualitative analysis began. 

The following sections describe the instrumentation, data collection, data analysis, and 
validity and reliability information for each of the four major techniques utilized during the site 
visits: surveys, interview protocols, classroom observations, and other instruments. 
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Surveys 

Instrumentation. Four survey instruments were developed, tested, and utilized in the ESS 
pilot test and employed during the site visits of this evaluation. The four target groups included ESS 
teachers, non-ESS teachers, ESS students, and parents. Each survey is briefly described below. 

• ESS teacher: This survey contained 24 questions; 18 items were selected-response and focused 
on demographics, attributes of the ESS program, recruitment procedures, teacher hiring and 
staff development, communication strategies, major outcomes, forces that help or hinder 
implementation, and overall effectiveness of the program. Six open-ended items asked for 
information related to strengths, weaknesses, and recommendations for improvements. This 
survey was administered at school to the full population of ESS teachers. 

• Non-ESS teacher: This survey contained 19 questions: 13 selected-response and 6 open- 
ended items, all similar to those on the ESS teacher survey. This survey was 
administered at school to the full population of teachers not engaged in the ESS program. 

• Parent of ESS student: This survey contained 1 1 questions. The initial 7 items were 
selected-response and focused on their children’s performance in ESS. Four open-ended 
items focused on best features or problems of the program and an explanation of why 
their children would or would not participate the following year. This survey was 
administered to parents of all students participating in the ESS program who had returned 
a signed parental consent fonn. One copy of the survey was sent home with the student and 
was completed by one parent individually or by both collaboratively. 

• ESS student: This survey contained 16 questions. Thirteen of the items were selected- 
response and focused on demographics, subjects studied in ESS, and a series of questions 
about academic behaviors and attitudes with which students agreed or disagreed. Three 
open-ended items focused on what students liked best about the ESS program and what 
changes should be made to the program. This survey was administered at school to all 
students participating in the ESS program who had returned a signed parental consent 
form authorizing their participation in evaluation activities. 

Data collection. During each site visit, the data collectors worked with the school 
coordinator to administer the four surveys. Surveys were generally distributed during the first 
day of the visit, and returned by the last day. There was wide variance among the participation 
rate of students, due to the fluctuating percentages of signed consent forms. 

Data analyses. After all of the fall/winter site visits were completed, AEL staff designed 
data entry templates using Remark optical scanning software. Surveys were scanned and then 
exported to SPSS for statistical analysis, including descriptive statistics. Comments were typed 
into the SPSS data files. Summer site visit data were scanned as they became available. 
Individual school files were merged into one master file by type of survey before analyses began. 
A breakdown of the number of surveys completed during the site visits is provided in Table 2. 
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Table 2: Number of Site Visit Survey Respondents by Role Group 



Group 


Fall/Winter 2001-02 


Summer 2002 


Totals 


ESS teachers 


175 


50 


225 


Non-ESS teachers 


297 


NA 


297 


ESS students 


775 


445 


1,220 


Parents of ESS students 


360 


216 


576 


Totals 


1,607 


711 


2,318 



Validity and reliability. These surveys were developed, tested, and utilized in the pilot test by 
Nesselrodt and Schaffer (2000a, 2000b). The instruments possess face and content validity and have 
proven their utility in a prior administration. To assess the degree of internal consistency reliability, 
Cronbach alpha coefficients were computed for this administration of the selected-response items on 
the four surveys (excluding demographic-type items and, for the parent survey, one particular multiple- 
response item). This administration of the ESS teacher survey resulted in a coefficient of .60; the non- 
ESS teacher survey, .53; the parent survey, .53; and the student survey, .70. 



Interview Protocols 

Instrumentation. Five interview protocols were developed, tested, and utilized in the 
pilot test and employed in this evaluation. The five target groups included district administrators 
(coordinators), school administrators (coordinators), ESS teachers, ESS students, and parents of 
ESS students. Each protocol is described briefly below. 

• District and school coordinators: These protocols each contained 20 questions and 

focused on aspects such as the history of the ESS program; a description of the services 
provided under the current program; how students are identified and referred to the 
program; recruitment and staff development for administrators and teachers; 
communication among teaching staff, students, and parents; processes for setting goals 
and monitoring student progress; and assessment and exit procedures. 

• ESS teacher: This protocol contained 16 questions and focused on aspects such as a 
description of the current ESS program, recruitment and staff development practices, 
communications with students and parents, curriculum and methodologies used in regular 
and ESS classrooms, and major strengths and weaknesses of the current ESS program. 

• Parent of ESS student: This interview protocol contained 1 1 questions and focused on 
perceived effectiveness of the program, areas of potential change, and perceived growth 
or success of their children. 

• ESS student: This protocol contained 6 main questions, with multiple sub-items within 4 
of the main questions and focused on perceived effectiveness of the program, areas of 
potential change, and perceived growth or success. 
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Data collection. Arrangements were made to interview the district coordinator at either 
the district office or the school during each site visit. With the exception of students, all 
interviews were conducted on an individual basis. Extensive interview notes were taken, which 
were later transcribed for analysis purposes. Students, parents, and teachers were randomly 
selected for interviewing. For students, a random selection was made using the signed parental 
consent forms received by the school coordinator. 

Data analyses. By the end of the summer site visits, all interviews had been completed. 
A breakdown of the number of interviews completed during the fall/winter and summer site 
visits is provided in Table 3. Typed interview responses were analyzed by questions, first for 
categories then for emergent themes. Next, data displays were developed for each question. 



Table 3: Number of Site Visit Interview Participants by Role Group 



Group 


Fall/Winter 2001-02 


Summer 2002 


Totals 


District coordinators 


15* 


0 


15 


School coordinators 


17 ** 


6 


23 


ESS teachers 


72 


26 


98 


ESS students 


74 


35 


109 


Parents of ESS students 


36 


13 


49 


Totals 


214 


80 


294 



* Three of the district coordinators had two schools within the 18 schools selected for site visits. 
**One of the school coordinators was unavailable due to an injury suffered the day before the 
site visit. 



Validity and reliability. These protocols were developed, tested, and utilized in the pilot 
test by Nesselrodt and Schaffer (2000a, 2000b). These protocols possess face and content validity 
and have proven their utility in a prior administration. For reliability, interviewers’ use of protocols 
at the training session and during the site visits established a satisfactory level of agreement. 



Classroom Observation Instruments 

Instrumentation. For the collection of regular classroom and ESS session data, three 
instruments were selected and employed in the pilot test and used in this evaluation. All three 
were developed and employed in prior research and evaluation studies, refined by AEF and KDE 
staff, and converted to a scannable format. The three instruments comprise the Special Strategies 
Observation System (SSOS), which is designed for use in a variety of settings to systematically 
collect data on essential elements of classroom behavior related to instruction, management, and 
context. The SSOS is a viable instrument for school effectiveness research due to its strong 
grounding in the current literature on effective teaching and its utilization of a variety of 
methodologies. This combination of instruments generates low-, moderate-, and high-inference 
data; this triangulation of information further documents the veracity of the data collected. Each 
instrument that makes up the SSOS is described below. 
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• Classroom Observation Form (COF): The COF is a combination observation system 
that is best described as a category system, with low inference items, and includes 
multiple coding procedures (Nesselrodt & Schaffer, 1993; Sullivan & Meehan, 1983). It 
is based on the Classroom Activity Record designed by Everston and Burry (1989) and 
the Stallings Observation System (Stallings, 1980). The top page of the COF collects 
typical demographic information, such as the school, observer, date, number of adults and 
students in class, subject being observed, and type of class (ESS or regular). The 
observations occur over 56 minutes, during which the observer switches between coding 
the entire classroom and focusing on a single student previously selected. Each of seven 
pages corresponds to eight minutes of class time. The first minute per page — the 
“snapshot” — looks at student engagement (i.e., the number of students on task, off task, 
out of the room, or waiting) and grouping strategies (i.e., whether clustered in teacher, 
aide, or student groups and type of involvement, such as working alone, management, 
interaction, or socialization). The remaining seven minutes per page focus specifically on 
the target student, and include coding one of 27 discrete activities for each minute. 

• QAIT assessment of classroom: This instrument is best described as a moderate and 
high-inference, simple coding, rating device. QAIT stands for Quality of Instruction, 
Appropriate Level of Instruction, Incentive, and Use of Time. Fitting on two sheets, it 
contains 40 items grouped under the four major categories. Each item uses a Likert- type 
rating scale of 1 to 5 ( unlike this class to like this class). This instrument was to be 
completed at the end of each observation session. 

• Classroom Environment and Resources Checklist (CERC): The CERC is a low- 

inference, simple coding, sign system. Printed on the front of one sheet, it contains 12 
classroom attributes that are coded either as present or not present, such as adequate 
lighting, use of multi-racial materials, posted assignments, etc. Next, 18 classroom 
resource items, such as textbooks, computers, and worksheets are listed; observers 
indicate whether such resources are visible or not. If they are, observers indicate whether 
they are used during the observation. This instrument also was to be completed at the 
end of each observation session. 

Data collection. Data collectors utilized these forms during the school site visits. The 
classroom observation segments were completed during the observation; QAIT and CERC forms 
were completed as soon after the observation as feasible. Only ESS observations were 
completed during the summer visits, as regular school was not in session. 

Data analyses. After the fall/winter site visits were completed, AEL staff designed data 
entry templates using Remark scanning software. SSOS data were scanned by school; data files 
were then cleaned and exported to SPSS for statistical analyses. School files were merged into 
one master file before analyses began. A total of 193 observations was completed during the 
fall/winter school visits and 20 were completed during the summer visits for a grand total of 213 
observations. 

Classroom observation data were averaged across the number of eight-minute intervals 
per each observation. Percentages of time for the classroom snapshots and target student 
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activities were calculated for both the regular classroom and ESS sessions. Data were analyzed 
using the 27 individual categories and by grouping these into four main categories: teacher-led, 
student-led, management/organization, and off- task. Student engagement data and time spent by 
the target student in the four main instructional categories also were analyzed by ESS 
implementation patterns. 

QAIT data were analyzed by creating four main subscales composed of the 40 individual 
items. Descriptive statistics were used to describe results for both regular classrooms and ESS 
sessions. Further, independent t tests were conducted to determine if a statistically significant 
difference existed between the two types of classes for each of four categories: quality of 
instruction, appropriate level of instruction, incentive, and use of time. 

CERC data were analyzed by calculating frequency percentages showing whether the 
classroom attributes were present in the regular classroom and ESS sessions. As well, frequency 
percentages were calculated to show whether various classroom resources were visible and used 
during the observations. 

Validity and reliability. These instruments were tested and utilized in the pilot test by 
Nesselrodt and Schaffer (2000a, 2000b). Thus, these instruments possess face and content 
validity and have proven their utility in prior research. A high degree of inter-rater reliability 
was achieved among the data collectors, given that every participant passed at or above the 85% 
criterion of the COF coding assessment held at the conclusion of the training session. To assess 
the degree of internal consistency reliability, Cronbach alpha coefficients were computed for this 
administration of the COF and QAIT instruments, excluding demographic-type items; this 
procedure was not appropriate for the CERC instrument, given its lack of variance in response 
options of either selected or not selected. For the COF instrument, this administration of the 
grouping strategy items resulted in a coefficient of .54; for the student engagement items, .82. 
For the QAIT instrument, this administration of all items resulted in a coefficient of .94; by 
subscale, the coefficients were .91 for quality of instruction, .74 for appropriate level of 
instruction, .88 for incentives, and .80 for use of time. 



Other Instruments 

Instrumentation. The School and Program Description Form was used in the pilot test 
and then refined by KDE and AEL staff for use in this evaluation. The form is machine 
scannable and fits on one sheet. It contains 15 items, 13 of which are demographic in nature, i.e., 
school characteristics, student enrollment, number of students and teachers involved in the ESS 
program, hours of operation, etc. The two open-ended items ask for a description of the major 
components of the ESS program and any unique characteristics of the school or community. 

The Innovation Component Configuration Map for Extended School Services (ICCM) 
was developed by the Kentucky Institute for Education Research (KIER, n.d.) in the mid-1990s. 
The ICCM is based on the Concerns-Based Adoption Model (CBAM) originated and developed 
by Gene Hall, Shirley Hord, and others (Hall & Hord, 1987; Hord, Rutherford, Huling- Austin, & 
Hall, 1987). Basically, the ICCM is a map depicting the 15 major components of the ESS 
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program in the three broad areas of student eligibility, school level program design, and district- 
wide ESS program planning. For each major component, there are three or four possible 
implementation variations (coded as 1 to 3 or 4, as appropriate), which were summed to create a 
total implementation score. The appropriate implementation levels for each component are 
determined through an analysis of all relevant data gathered during the site visit. When 
completed, a picture of the ESS component configurations was established based on the 
implementation scores. When the ICCMs for all the site visits were completed, a study of their 
various patterns of implementation was conducted; patterns were then used in conjunction with 
other variables to determine whether statistically significant differences occurred among the 
identified implementation patterns. 

Data collection. The School and Program Description Forms were to be completed and 
gathered during each of the 24 site visits. However, only 22 of these forms were returned to 
AEL for analysis. An ICCM form was completed by the data collectors for each school site visit 
and returned to AEL. 

Data analyses. A data entry template was developed for each instrument using Remark 
scanning software. After the instruments were scanned, the files were cleaned and then exported 
to SPSS for analysis. Descriptive statistics were generated for the School and Program 
Description Form items; the two open-ended items were qualitatively analyzed by common 
themes. Descriptive statistics were generated for the 15 ESS components on the ICCM. 
Further, AEL staff met to visually examine the ICCMs to detect patterns of implementation. 
Based on this discussion, the only patterns emerging were based on the summed score for each 
school, which could range from 15 to 47 (the greater the score, the greater the degree of 
implementation). These patterns were used to correlate with a number of other school-level 
variables, including proficiency and accountability levels, attendance and retention rates, number 
of parent volunteer hours, average years of teaching experience, expenditures per student, and 
number of drug/weapon/assault incidents. These patterns also were used to determine 
statistically significant differences for student engagement and instructional activity categories 
from the SSOS, selected key items from ESS coordinator and teacher surveys, and the AEL 
CSIQ scales. 

Validity and reliability. The School and Program Description Form was developed, 
tested, and utilized in the pilot test by Nesselrodt and Schaffer (2000a, 2000b). The ICCM is 
based on solid research endeavors such as the Concerns-Based Adoption Model. Thus, these 
instruments possess face and content validity and have proven their utility in a prior 
administration. To assess the degree of internal consistency reliability, Cronbach alpha 
coefficients were computed for this administration of the ICCM; this procedure was not 
appropriate for the School and Program Description Form, given its strictly descriptive nature. 
This administration of all items in the ICCM instrument resulted in a coefficient of .82; by the 
three broad areas, the coefficients were .78 for student eligibility, .74 for school-level program 
design, and .44 for district-wide ESS program planning. 
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FINDINGS 



Findings are presented as comprehensive summaries of data points that are directly 
aligned with the five broad evaluation topics and their related subquestions. 



Evaluation Topic One: Identification, Referral, and Assignment of Services 

a. Why do students participate in ESS? 

Nearly all (95%) of the 151 district coordinators who responded to the district 
coordinator survey indicated that most students were referred to the program because they were 
in danger of failing and needed to improve their academic performance. In addition, 90% of the 
district coordinators interviewed (n = 15) indicated that the program was intended to help 
students succeed, or improve academically. In addition, 73% or more of both ESS (n = 225) and 
non-ESS (n = 297) teachers surveyed indicated that their students received ESS services to 
improve academic achievement and because their students were in danger of failing. 

Interviews with parents of students participating in ESS confirmed this finding, given that 
22% of the 49 parents who were interviewed indicated that their children were participating in the 
program because they needed to improve their grades in a specific subject and 7% needed to improve 
their grades in general. In addition, 19% of the ESS students interviewed indicated that they were 
participating in ESS because they needed to do better in school (7%), had failed a class (5%), needed 
to improve a specific grade (4%), or needed to graduate (3%). Nearly one third (31%) of these 
students participated because they volunteered, indicating a desire to improve grades, get extra help, 
or simply wanting to come to the program as specific reasons for their participation. 

b. How are students referred for ESS services and by whom? 

According to 87% of the 225 ESS teachers surveyed, teacher recommendations were a 
basis for selecting students for participation in ESS. Other means of selecting students for ESS 
included parent requests (56%), student requests (45%), standardized test scores (9%), and other 
(10%). Similarly, 82% of the non-ESS teachers surveyed indicated that selection was based on 
teacher recommendations, 44% indicated that it was based on parent requests, 33% chose student 
requests, 9% chose standardized test scores, and 8% selected other. Responses to the parent 
questionnaire paralleled these data, with 53% indicating teacher recommendation, 32% 
indicating student self-referral, and 29% indicating that parents referred the students. 

ESS teachers’ (n = 98) interview responses to the question about student selection were 
somewhat consistent with responses on their surveys. Thus, 24% of ESS teachers stated that teacher 
recommendations were used to identify students, 3% stated that school counselors selected students, 
and 2% stated that administrators referred students to the program. In addition, 16% of the ESS 
teachers responded that struggling learners were identified as needing ESS, 8% stated that test scores 
were used to identify students in need of ESS, and 18% and 14% of ESS teachers said selection was 
based on parents referrals and students’ self-referrals, respectively. 
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Thirty-one percent of school ESS coordinators who were interviewed (n = 23) said that 
referrals were made by teachers, 20% said they were made by parents, 10% said they were made 
by students, and 22% stated that eligibility was based on a variety of things including need and 
recommendations from professional school staff. Further, 25% of the 15 district ESS 
coordinators who were interviewed indicated that eligibility for ESS was determined by 
following district guidelines. Finally, when asked if there was a fonnal method of referral, 40% 
of the district level ESS coordinators interviewed indicated that teachers initiated referrals to 
which parents later consented. 

c. Once the referral has taken place, how are individual student goals determined? 

Establishing individual students’ goals for the program appeared to rely heavily on 
parents and teachers, with some student participation. According to 30% of school ESS 
coordinators who were interviewed, the regular or classroom teacher, who was often also the 
referring teacher, determined the goals for students when he/she identified the students’ needs or 
reasons for referral. Another 12% of the coordinators said that the regular and ESS teacher set 
the goals together, and 6% of the coordinators said that the goals were stated on the referral 
forms. Only 24% of the ESS school coordinators said that students were involved in setting the 
goals for their participation, with 12% involving the regular teacher and student working 
together, 6% involving the students and ESS teacher working together, and 6% involving the 
students working alone. Moreover, although parents weren’t identified by the ESS coordinators 
as being part of the goal-setting process, 58% of the parents who were interviewed stated that 
they were indeed part of this process via collaboration with a teacher, as part of planning for 
college, by closely monitoring their children’s progress in school, and by working with their 
child at home. 

Regardless of the apparent lack of participation of students in the goal-setting process, 
only 35% of the ESS students interviewed said that they participated in ESS because someone 
else wanted them to do so (17% of the students stated that their teachers suggested their 
involvement in ESS, and 18% indicated that one or both parents had encouraged it). Overall, it 
seemed that students’ goals were heavily influenced by their parents and/or teachers but 65% of 
the students apparently accepted these goals as their own and/or understood why they were 
expected to benefit from ESS. 



Evaluation Topic Two: Profiles of Students Receiving Services 

a. What are the grade levels of students receiving ESS? 

ESS students in the sample schools were asked to provide information about their grade 
levels. A total of 1,195 students replied to this query on the student survey. Nearly half (48%) of 
those responding indicated that they were in the high school grades: 5% in 9th, 14% in 10th, 
14% in 1 1th, and 15% in 12th. ESS students in the elementary grades constituted the next largest 
group (28%), with 3% in 1st, 6% in 2nd, 7% in 3rd, 6% in 4th, and 6% in 5th. Slightly fewer 
(25%) were in middle grades: 10% in 6th, 8% in 7th, and 7% in 8th. It is interesting to note the 
increases in ESS students in the 6th and 10th grades; these increases perhaps indicate that 
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students transitioning from the elementary and junior high schools may require the additional 
academic support offered via ESS. 

While nearly half of the students who returned surveys were at the high school level, this 
does not imply that a larger proportion of high school youth were participating in ESS sessions, 
but rather reflects the larger size of the high schools; further, half of the summer visits were 
conducted at the high school level. When comparing the percentage of ESS students at a school 
to the total student enrollment, overall building-level percentages were similar for elementary, 
middle, and high schools — from 16% to 19% for all three levels. 

b. What subgroups (gender, race, etc.) are represented by students receiving ESS? 

Gender data were available for students who completed the student survey in the sample 
school sites. Of the 1,163 students who responded to this demographic item, 52% (n = 603) were 
male and 48% (n = 560) were female. 

When disaggregated by school level, it appeared that the percentages of boys and girls 
attending ESS programs in the sample sites remained fairly stable, with boys’ participation 
increasing slightly at the middle school level. Boys constituted 49% of attendees in elementary 
schools, and girls, 51%. Middle school boys accounted for 53% of participants, and girls 
accounted for 47%. Likewise, 53% of ESS attendees were boys at the high school level, and 47% 
were girls. 

c. What are the achievement levels, as measured by grades and proficiency levels, of the 
students receiving ESS? 

According to 95% of the 151 district coordinators who replied to the survey mailed to all 
Kentucky district coordinators, most students were referred to the program because they were in 
danger of failing. Ninety-five percent also reported that a common reason why students were 
recommended for ESS was to improve their academic performance. Of the 837 school ESS 
coordinators who returned their surveys, 76% indicated that students were referred to the 
program because they were in danger of failing and 92% tended to report, similarly to district 
coordinators, that students were recommended to the program to improve their academic 
performance. 

Data gathered during site visits suggested that students referred to ESS were at some risk 
for failure, but not to the degree suggested by respondents to the district and school coordinator 
surveys. Of the 15 district coordinators interviewed during AEL site visits, for example, fewer 
than half (45%) reported that students in their districts were recommended for program 
participation because they were not succeeding academically. Similarly, only 43% of the 23 
school coordinators interviewed during AEL site visits thought that students were referred to 
ESS because their perfonnance was deficient (although an additional 6% reported that students 
were referred because they were not achieving to their potential). 

Interestingly, ESS teachers interviewed during AEL site visits reported more frequently 
that students were referred because they were struggling academically. ESS teachers were asked 
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“What are the main problems the program is intended to solve? What are the main 
goals/purposes of the program in your district?” Of the 98 ESS teachers who replied to this 
question, 23% reported that the program was used to assist struggling learners, 13% to improve 
student performance, another 13% to prevent failure or attrition, 7% to improve performance in 
specific academic subjects, and 4% to improve test scores. The 225 ESS teachers who 
completed an ESS teacher survey tended to corroborate this view of student performance. Asked 
to select from a list the most common reasons students receive ESS services, 84% indicated that 
they were to improve their academic achievement, and three fourths (75%) noted that students 
were in danger of failing. In addition, 11% reported that ESS students were in danger of 
dropping out of school. 

Non-ESS teachers likewise thought that ESS students were referred to the program 
because they were performing poorly. Nearly three fourths (73%) of the 297 non-ESS teachers 
responding to the survey reported that their students received ESS services because they were in 
danger of academic failure. Seventy-three percent also agreed that students were referred to ESS 
to improve their academic achievement. 

d. What are the characteristics of students receiving ESS that put them at risk of 
dropping out of school — e.g., low achievement in school, poverty, single-parent homes, etc.? 

Data from the School/Program Description Fonn revealed some characteristics that may 
put students at risk of dropping out of school. For instance, when asked to describe any unique 
characteristics of their community, school, or student population, respondents most frequently 
noted general issues such as socioeconomic status of students (16%) and rurality (16%); at a 
more specific level, respondents noted a high incidence of eligibility for free or reduced-price 
meals (13%) and diverse locales (10%). 



Evaluation Topic Three: Profiles of ESS Programs and Their Implementation Patterns 

a. What are the major components of the ESS program? 

The major components of the Extended School Services program were identified by 
Kentucky educators in the mid-1990s as part of a research project completed by the Kentucky 
Institute for Education Research (KIER). Through a series of meetings, discussions, drafts, and 
reviews, KIER staff guided teams of Kentucky educators in developing of a series of “innovation 
component configuration maps” for all the major thrusts in the KERA law. Specifically, the 
component map developed for the ESS program was titled the Innovation Component 
Configuration Map for Extended School Services, or ICCM for short. The ICCM depicts the 
major components of the ESS program in the three broad areas of Student Eligibility, School- 
Level Program Design, and District-Wide ESS Programming Planning. 

There were unequal numbers of major program components within the three broad areas 
named above. In the Student Eligibility area, the components included referral guidelines, 
student selection, entry and exit process, and student assessment for eligibility. In the School- 
Level Program Design area, the components included school transformation planning, 
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scheduling, staff selection, staffing patterns, instructional practices in ESS programs, organizing 
and grouping students, and instructional resources. Finally, within the District-Wide ESS 
Program Planning area, the major components included collaborative planning processes, 
program evaluation, fiscal management, and linkages with other KERA strands and other 
supporting programs. In sum, then, there were 15 major components in the ESS program, as 
derived from prior KIER research. 

Then, for each of the 15 major components of the ESS program, there was a series of 
possible implementation variations. There usually were three implementation variations for each 
component, but two components (instructional practices in ESS programs and program 
evaluation) had four possible variations. The component implementation variations were 
arranged from left to right on the page for each component and labeled as “Variation A” through 
“Variation C,” except for the two “Variation Ds.” These implementation variations were ordered 
from most ideal (Variation A) to least ideal (Variation D). The most ideal variation (A) was 
viewed by the ICCM developers as the implementation of that component that was the best that 
an ESS program could achieve. It follows, then, that the most ideal implementation of the ESS 
program would be one that was implementing all 15 major components at the “A” level. Of 
course, in practical terms, it would be very difficult for any ESS program to achieve this high 
standard. To discover where schools in the state were with their implementation of the ESS 
major components, a variety of evaluation study data sources were inspected. 

b. How does the implementation of the ESS program components vary by stakeholder 
groups? 

During site visit interviews, district ESS coordinators were asked to describe their ESS 
services so as to solicit responses, of a general nature, about their implementation of the major 
components of the ESS program. The 15 coordinators’ responses were grouped into 11 
categories, ranging from 43 to 2 percent of the replies. Comparing the response categories to the 
ESS major components and their implementation variations showed that the largest response 
category (days/times offered) was related to variations of the scheduling component in the 
School-Level Program Design area. This category included replies such as summer school, 
afternoon programs, Saturday school, a.m. and p.m. services, intersession/breaks, before school, 
night classes, and off campus. The category with the second-largest number of responses (12%) 
was labeled “monitor budget,” which relates most directly to variations of the fiscal management 
component in the District Wide ESS Program Planning area. 

Also during site visits, school ESS coordinators were asked the same question about ESS 
services. Their responses were placed into 13 categories, ranging from 20 to 3 percent of the 
replies. It is interesting to note that seven of those categories can be collapsed and related to 
variations of the instructional practices in ESS program component in the School-Level Program 
Design area. Totaling 63% of the school coordinators’ responses, the seven categories were 
subject/content areas, remediation/skills help, tutorial help, test preparation, writing/portfolio, 
homework, and higher-order skills. 

During the site visits, 98 ESS teachers were interviewed and asked several questions 
about the implementation of key elements in their schools, specifically pertaining to variations of 
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implementation of major components in the area of School-Level Program Design. When asked 
about time of day for their ESS program, 52% of the teachers replied after school, 26% said 
summer school, 11% said morning before school, 6% replied Saturday, 3% said evening hours, 
and the remaining 3% said the times were flexible. When asked to describe the staffing of their 
programs, 30% of the teachers’ responses fit into a category of elective/volunteer. Teachers 
were asked to state the number of students per class: 30% of the responses were in a category of 
6-10 students to 1 teacher, 17% were in 2-5 to 1, 17% in 11-15 to 1, 15% in 4-30 to 1. When 
asked about the curriculum of their ESS program, the teachers’ responses fit into 10 categories: 
the top 3 categories tied with 18% of the responses and included everything, same as regular 
classroom; reading; and math. When asked to describe the key elements of the instructional 
methods of their ESS programs, teachers provided a wide variety of responses fitting into 18 
categories. At 28%, the category with the most responses was individualized instruction, which 
was followed by the category of small groups at 13%. Finally, the ESS teachers were asked to 
describe their adaptations to student’s needs. Thirty-five percent of the teachers’ responses fit 
into the category of individualized instruction. Another 16% were in a category labeled learner 
needs (but not individualized). 

Also, as part of the site visit data collection, 49 parents of ESS students were interviewed. 
One question they were asked was “What are the best parts of ESS for your child?” The parents’ 
responses to this question provided some additional stakeholder infonnation about the variations of 
implementations of the major components in the ICCM. Almost one third (32%) of the parents’ 
responses fit into a category of individual attention. At 15%, the second largest category dealt with 
the positive outcomes of the program on the academic standing of their children. 

During the site visits, 109 students enrolled in the program were interviewed and asked 
several questions relating to the implementation variations of the major components. For 
example, related to the component of entry and exit process in the Student Eligibility area, 
students were asked how long they had been in ESS and what subjects they studied. Student 
responses to these two questions were often combined rather than separate, but four categories of 
responses were specific to the length of time in ESS. The category with the largest number of 
responses at 21% was called the first semester or first year; then, 9% responded with second 
year. In the area of School-Level Program Design, students were asked if their ESS teacher was 
the same as their regular teacher. Fifty-six of the responses were in a category of not the same 
teacher, while 37% were in the same teacher category, and 5% said it was mixed (different for 
one subject, same for another). 

ESS students were asked “What do you learn about in ESS?” The responses helped to 
describe the variations of instructional practices and instructional resources in the School-Level 
Program Design area. Student responses to this question were categorized into 15 groups, with 
some naming a subject and others naming an instructional practice or resource. Students were 
asked to describe what they did in their ESS classes that was different or the same as their 
regular classes: 86% of the responses were classified into 13 difference categories. Students 
were asked if the ESS teachers did anything special to teach them in ESS class that their regular 
teachers didn’t and, if so, what. Seventy-nine percent of the responses (79%) were classified into 
10 categories of different, special things, including more individual help (34%). 
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c. What are the patterns of implementation of the ESS components for the more 
effective ESS users and for the less effective ESS users? 

The patterns of implementation of the ESS components were determined through an 
analysis of the ICCM forms completed by the data collection teams at the conclusion of the 24 
site visits. There were 15 major ESS components on the ICCM; each had at least three possible 
variations (A-C) and two had four possible variations (A-D). For analysis purposes, each 
variation was assigned one point and the points per each completed ICCM were totaled. The 
total scores could range from a minimum of 15 to a maximum of 47; the higher the point value, 
the higher the level of ESS implementation toward the most ideal implementation of the most 
components. (Higher implementation scores often are called “high-fidelity implementations.”) 

Actual scores on the ICCM forms ranged from a low of 24 to a high of 45 on the 47-point 
scale. Four scores were in the 40s, 15 in the 30s, and 5 in the 20s. To determine the patterns of 
implementation within these scores, the total implementation scores and the actual variation 
scores were examined simultaneously to discern similar groupings or patterns. Four overlapping 
patterns of implementation emerged from this analysis: Pattern 1, scores of 24 to 31 (n = 8); 
Pattern 2, scores of 33 to 38 ( n = 7); Pattern 3, scores of 39 (n = 5); and Pattern 4, scores of 40 to 
45 ( n = 4). Given the nature of these patterns, a traditional graphic “map” was not generated. 

The scores for Pattern 3 were very close to the scores for Pattern 4 and, likewise, the 
scores for some of Pattern 2 were close to the scores of Pattern 3. The distinguishing 
characteristic of the groups, then, rests in the details of the patterns of the implementations of the 
15 major ESS components. Put another way, although the total implementation scores were 
close, the patterns of implementation of the various components differed. For example, Pattern 4 
was the most effective — the ESS implementation with the most fidelity to the ICCM. Pattern 4 
implemented all components at variation A or B except for a single school that had all A and B 
variations plus two components with a C variation. In Pattern 2, the second-most effective ESS 
user group, all schools had implementation patterns consisting of all A and B variations and just 
one C variation. The implementation of components for Pattern 3, next-to-lowest effective users, 
consisted of some A and B variations plus either two or three C variations. Last, Pattern 1, the 
least effective ESS users, had implementation patterns either of some A and B variations plus 
four or more C variations or some A, B, C, and D variations. 



Evaluation Topic Four: Services to Students Placed at Risk 

a. Are ESS programs serving students placed most at risk academically? 

One method to determine whether ESS is serving students placed most at risk 
academically is to examine how students are referred to ESS. Permitting multiple “paths” to 
ESS increases the likelihood that all children in need of ESS services will be identified. Teacher 
recommendation was used in all districts as a selection method. In two thirds of the districts, 
parent requests also served as a route into ESS. In 45% of the districts, students could request 
ESS services. In a third of the districts, standardized test scores were used to detennine ESS 
eligibility. 
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There are myriad reasons why a student might be considered at risk for failure, with the 
most obvious signal being poor performance in a subject. The main reasons students received 
ESS services were because they were in danger of failing (mentioned by 95% of district and 76% 
of school ESS coordinators), and/or needed to improve academic perfonnance (noted by 95% of 
district and 92% of school ESS coordinators). 

More than 88% of all teachers, school coordinators, and district coordinators who were 
surveyed reported increased academic achievement as a main outcome of the ESS program. Of 
the 576 parents who answered the survey item, 58% said that their children had improved 
understanding of the subject material, 36% reported that their children were passing the subject, 
and 22% remarked that their children were passing the grade as a result of participation in ESS. 

b. Are ESS programs meeting the needs of students placed at risk academically? 

Questions were asked in several of the data collection activities to answer this evaluation 
question. The 49 parents of ESS students who were interviewed during site visits were queried 
about their perceptions of ESS effectiveness. When asked how their children were doing in 
school since participating in ESS, more than half (54%) responded that their children’s 
performance in school had improved, with comments such as “He has improved,” and the child 
had “brought grades up.” Seventeen percent of the parents believed that the instruction and 
assistance their children received in ESS were helpful, saying, “They work better,” and that the 
child was “able to do homework.” When asked what they thought their children had gained from 
the ESS program, 23% of the 49 parents commented that their children were giving school more 
attention because school was giving the children more attention, i.e., one-on-one instruction and 
tutoring. Better grades were a noticeable outcome mentioned by 15% of the parents interviewed. 
Fourteen percent of parents had witnessed increased self-esteem in their children. Ten percent of 
the parents believed their children’s participation in the ESS program had resulted in better 
homework skills; an additional 9% of parents had noticed their children having better general 
study skills and organizational skills. 

A wider cross section of parents of ESS students was administered surveys. One 
question asked parents how their children’s performance in school had changed since they began 
participating in ESS. Parents were provided five answer choices, ranging from much better to 
much worse. Of the 565 parents who answered the question, 21% said they thought their 
children were performing much better in school since participating in ESS. More than half of the 
parents (57%) said their children were doing better in school. 

Both ESS and non-ESS teachers were asked in the teacher surveys what the most 
important ESS outcomes were for students. The teachers could provide more than one response. 
Almost all (95% of the ESS and 89% of the non-ESS teachers) reported enhanced academic 
achievement. Two thirds (65%) of the ESS teachers observed increased motivation on the part 
of students; in comparison, 38% of the non-ESS teachers noted increased motivation as an 
outcome. Sixty percent of the ESS teachers and 29% of the non-ESS teachers indicated that 
students had better self-esteem because of participation in ESS. 
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District and school ESS coordinators were asked the same question about student 
outcomes. Almost all (99%) of the district coordinators stated enhanced academic achievement 
as an outcome of the program. Approximately two thirds (62%) of the district coordinators 
marked increased motivation as an outcome, and 48% reported increased student self-esteem. 
School coordinator responses were similar, with 98% noting enhanced academic achievement, 
70% reporting increased student motivation, and 56% noting an increase in student self-esteem. 

On the student survey was the statement, “I am a better student this year,” to which 
students either agreed or disagreed. More than three fourths of the 1,201 students (78%) 
responded that they were better students this year. Eighty-five percent of students agreed that the 
ESS program was helping them this year, and 86% stated that they asked for help in ESS when 
they needed it. Finally, when asked if they liked school, 67% responded affirmatively. 

c. Are ESS programs identifying and addressing those factors that place students at risk 
for failure? 

There are challenges in working with students placed at risk academically. A quarter of 
the ESS teachers who were surveyed commented that motivating students and getting students to 
attend ESS were hurdles they faced (“Motivating students while focusing on academic 
challenges”). Attendance and student motivation were also mentioned by non-ESS teachers 
(22%), school ESS coordinators (19%), and district coordinators (12%). 

The emphasis at most schools appeared to be individualized instruction, or tutoring. This 
type of instruction was helping students who needed extra time and assistance to master material. 
Students were highly appreciative of the extra attention from teachers that ESS provided. 
Eighty-five percent of those surveyed indicated that ESS was helping them that school year, and 
39% of ESS students who were interviewed commented that the individual tutoring was helping 
them. The additional time to do make-up work or redo work or tests on which the student 
received a poor grade also were mentioned by 31% of the students who responded to the survey 
as the aspect they liked best about the ESS program. 



Evaluation Topic Five: ESS Implementation Patterns and Outcomes 

a: How does the fidelity of ESS implementation correlate with academic index scores? 

As noted under evaluation question three, four distinct patterns of ESS implementation 
emerged from the ICCM instrument. These implementation scores were used to determine 
Pearson correlation values with academic index scores from the 2000-2001 Kentucky Core 
Content Tests (Kentucky Department of Education, 2002) for the following subjects: reading, 
science, mathematics, writing, social studies, arts/humanities, and practical living/vocational science. 
Table 4 presents correlation values of ESS implementation with each of the above subjects. 

As can be seen in Table 4, correlation values between the ICCM total implementation scores 
and each of the school-level variables were very small, indicating a lack of relationship between 
them. None was significant, which was not unexpected given the small sample size of scores. 
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b: How does the fidelity of ESS implementation correlate with school-level variables such 
as retention, discipline, attendance, etc. ? 

Again utilizing the ICCM total implementation scores, Pearson correlations were 
generated for a number of school-level variables, including attendance rate, retention rate, 
average years of teaching experience, the number of drug/weapon/assault incidents, the spending 
amount per student, and the number of parent volunteer hours (see Table 4). Again, these data 
were culled from the school report cards provided on the Internet (Kentucky Department of 
Education, 2002). As noted above, correlation values between the ICCM total implementation 
scores and each of the school-level variables were very small, indicating a lack of relationship 
between them. None of the correlations was statistically significant. 



Table 4: ICCM Implementation Score Correlations With School Variables 



School Variables 


Pearson Correlation 
with ICCM Total 
Implementation Score 


KCCT* Reading - Academic Index 


-.044 


KCCT Science - Academic Index 


-.067 


KCCT Mathematics - Academic Index 


-.010 


KCCT Writing - Academic Index 


.039 


KCCT Social Studies - Academic Index 


-.234 


KCCT Arts/Humanities - Academic Index 


-.018 


KCCT PL/VS - Academic Index 


-.015 


Average Years of Teaching Experience 


.021 


Number of Parent Volunteer Hours 


.077 


Spending Amount per Student 


.233 


Attendance Rate 


.147 


Retention Rate 


-.028 


Number of Drug, Weapon, or Assault Incidents 


.029 



* Kentucky Core Content Test 



Note : ICCM scores could range from 15 to 47; the higher the score, the higher the level of ESS 
implementation; actual scores ranged from 24 to 45. 



c. How does the fidelity of ESS implementation distinguish between schools with 
minimum and maximum achievement gaps? 

Each of the site visit schools was identified by KDE staff as having either a minimum or 
maximum achievement gap in overall academic index scores between White and minority 
students. To determine whether the ICCM implementation score could differentiate between 
these two classifications, an independent t test was conducted. The mean ICCM implementation 
score for the 13 schools classified as minimum gap was 36.54 (standard deviation of 5.25); the 
score for the 11 maximum gap schools was 32.45 (standard deviation of 5.97); the mean 
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difference was 4.08. With a t value of 1.78, and a significance of .088, this difference 
approached but did not reach statistical significance at the .05 level. 

d: What are the similarities and/or differences among ESS models (i.e., implementation 
pattern) within the classroom observations? 

For the ESS classroom observations, eight-minute segments were averaged into one set 
of scores per observation, then aggregated and classified by school into the four patterns of ESS 
implementation (lowest to highest levels of implementation) as defined by the ICCM instrument. 
For the classroom snapshot of the observation, students were coded during a one-minute 
observation as being on task, off task, out of the room, or waiting. One-way analyses of variance 
(ANOVAs) were conducted to determine whether statistically significant differences occurred 
among the mean number of students in each of the activities by implementation pattern. No 
significant differences were found between any of the pattern groups. For the number of on-task 
students, means ranged from 8 to 10 across patterns (standard deviations of 3 to 4); for the other 
three activities, means were all below 1 (standard deviations of 0 to 1). 

Then, attention was given to the target student segment of the ESS classroom 
observations. Again, eight-minute segments were averaged into one set of scores per 
observation, then were aggregated and classified by school into the four patterns of ESS 
implementation. The 27 individual activities that a target student could be involved in were 
grouped into the four main categories of teacher-led, management/organization, student-led, and 
off-task. One-way ANOVAs were conducted to determine whether statistically significant 
differences occurred among the mean number of minutes spent in each of the categories by 
implementation pattern. While no significant differences were found between any of the pattern 
groups, there were fluctuations across patterns. 

Time spent in teacher-led activities ranged from an average of 21 minutes in Pattern 4 
(highest level of ESS implementation) to 30 minutes in Pattern 3 (second highest level of 
implementation), with standard deviations of 3.46 and 16.44, respectively. Time spent in 
student-led activities occurred most often in Pattern 4 with a mean of 25 minutes (standard 
deviation of 7. 15), and least often for 16 minutes in Pattern 3 (standard deviation of 8.72). 

e. What are the similarities and/or differences among ESS models (i.e., implementation 
pattern) within selected data measures? 

The following analyses are further investigations of what factors might be underlying the 
four different patterns of ESS implementation. In particular, attention was given to inspecting 
forces that led to successful ESS implementation and barriers that hindered implementation. 
Available data from the 24 school coordinators from the statewide survey administration were 
utilized for this analysis. Further, data from ESS teachers’ surveys during the 24 site visits were 
aggregated by school and classified into four patterns of ESS implementation. The seven forces 
included (1) clear support or mandate from district or other political actions, (2) clear support from 
parents or community, (3) additional financial support, (4) excellent staff development and follow- 
up, (5) excellent relationships among staff, (6) outstanding administration, and (7) other. 
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All ESS teachers identified five of the seven forces as aiding their ESS implementation: 
district support, community support, financial support, staff relationships, and administration. 
Interestingly, district support was the only force that teachers within schools in Pattern 4 (highest 
level of implementation) selected more often than teachers at other schools; Pattern 4 had 50%, 
Pattern 1 (lowest level of implementation) had 38%. For the other four forces, ESS teachers in 
Pattern 1 showed higher percentages than Pattern 4 (differences of at least 25%), indicating a 
greater perception that these forces were prevalent in their schools. In comparison, school 
coordinators identified the above forces, along with staff development. Data showed two forces 
(district support and staff relationships) with large differences between Patterns 1 and 4 (75% 
difference in district support and 25% in staff relationships). 

ESS teachers selected only two problems (inadequate financial support and student 
transportation) from a list of 10 options on the survey. A fourth of teachers within Patterns 1 and 
4 each selected inadequate financial support; no Pattern 2 or 3 teachers selected this problem. 
All teachers indicated student transportation was a problem; however, Patterns 1 and 4 were 
equal at 25%. In comparison, school coordinators selected four problem areas: inadequate 
financial support, student transportation, student opposition, and other. For Pattern 4, 100% of 
the coordinators indicated inadequate financial support, compared to 13% of the Pattern 1 
coordinators. Similarly, more Pattern 4 coordinators indicated some other reason was causing 
problems (33%), compared to 13% of the Pattern 1 coordinators. For student transportation, the 
trend reversed, with 75% of the Pattern 1 coordinators indicating this was a problem, compared 
to 33% of the Pattern 4 coordinators. Similarly, more Pattern 1 coordinators indicated student 
opposition (50%), compared to Pattern 4 coordinators (33%). 

Next, data resulting from the administration of the AEL CSIQ instrument were 
aggregated by school and classified by ESS implementation pattern for the site visit schools to 
determine whether statistically significant differences in scale scores occurred between patterns. 
One-way ANOVAs were generated for each of the six scales by the four patterns of ESS 
implementation. One statistically significant difference was found in the School/Family/ 
Community Connections scale (F(3,20) = 3.39, p < .05). Tukey’s HSD was used to pinpoint 
which patterns differed significantly and revealed that only Pattern 4 (highest implementation 
level) differed from Pattern 3 (second highest implementation level). Pattern 3 had a mean score 
of 50.57 (standard deviation of 1.97), compared to a mean for Pattern 4 of 43.32 (standard 
deviation of 4.15) for a mean difference of 7.25. There is a large effect size associated with this 
difference (-2.23), indicating that there is not only a statistically significant difference, but also a 
meaningful one in a practical sense. While only one significant difference was found, there were 
slight fluctuations among all six of the scales. 

Finally, Pearson correlations were generated for the six mean AEL CSIQ scale scores of 
the site visit schools with the ICCM total implementation score. The correlations follow: 
Learning Culture, .137; School/Family/Community Connections, -.070; Shared Leadership, .046; 
Shared Goals for Learning, .037; Purposeful Student Assessment, -.031; and Effective Teaching, 
.211. These correlations were all very small, indicating a lack of relationship between the scales 
and the implementation score. None of the correlations was statistically significant. 
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CONCLUSIONS 



A number of conclusions were drawn from the findings presented within the 
comprehensive evaluation of the statewide Kentucky Extended School Services program. These 
conclusions are organized by nine topical areas. 

Student Demographics 

• In terms of the proportion of ESS enrollment to student enrollment, ESS participation is 
fairly equal across elementary, middle, and high school building levels; however, 
participation varies widely at the individual school level. 

• The participation of boys and girls in ESS is roughly equivalent, particularly at the 
elementary level. However, fewer females participate in the program at the middle and 
secondary levels. This warrants further investigation to detennine whether middle and high 
school girls need fewer ESS services or if they are simply less interested than boys in ESS 
participation. 

• Students attending ESS programs are characterized by coming from poorer areas (rural 
and inner city), which lack resources. These circumstances place students at risk of 
academic failure and dropping out of school. 

Adherence to Intended Goals 

• Generally, students are referred to ESS because they are not perfonning well academically 
and may be in danger of failing. Other reasons noted were to extend learning time, sustain 
current levels of perfonnance, or improve self-esteem. Some students take advantage of ESS 
services because they are in jeopardy of failing at least one class/subject. Thus the 
achievement of most ESS students is depressed when they first begin participating in the 
program. 

• There is a great deal of consistency among the perceptions of coordinators, teachers, and 
parents as to how students are referred to ESS; the majority believe that students are 
referred most often by classroom teachers. However, students report that they most often 
self-select into the program. It may be that students are taking credit for self-selection by 
agreeing to participate in this voluntary program after a teacher or parent has made the 
suggestion. Either approach seems to allow enough flexibility for the intended 
population to become involved with the program. 

• The students’ regular teachers, ESS school coordinators, and ESS teachers most often 
determine individual student goals, with parents and students themselves being involved 
to a lesser extent. Thus students’ goals appear to be heavily influenced by their teachers, 
yet the majority of students adopt these goals as their own and appear to understand why 
they are expected to benefit from participation in the ESS program. 
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• There is congruence among perceptions of the intended and actual outcomes of the ESS 
program. All stakeholders agree that the ESS program is helping students increase their 
academic achievement, pass courses and grades, and decrease school failure. 

• Given the main reasons for referral, and the outcomes perceived by respondents, it is 
evident that the ESS program is operating within its framework and addressing the main 
goals it is intended to accomplish. 

Classroom Instruction 

• ESS and regular classrooms differ on two major dimensions: quality of instruction and 
appropriate level of instruction. Quality of instruction is better in regular classrooms, but 
instructional level is more often appropriate in ESS classrooms. 

• ESS classrooms tend to engage in student-led activities, often involving independent 
seatwork and pair seatwork. Thus a “typical” ESS classroom appears to be one in which 
students work independently, receiving individualized instruction as needed. A strength 
of the ESS classroom arrangement is that students are receiving the one-on-one tutoring 
they need and have the opportunity to have concepts not mastered retaught to them. 

• While computers are almost universally available in both ESS and regular classrooms, 
very limited use was made of this resource. However, the environmental checklist did 
not differentiate between one or multiple computers, so in classrooms with only a single 
computer, usage may be restricted to teacher purposes. 

Student Outcomes 

• The ESS program appears to be having an impact on student performance. Nearly all 
teachers and coordinators indicate that participation in ESS has led to increased academic 
achievement. Further, parents report increased understanding of subject material by their 
children, that their children are passing a particular subject, or that their children are now 
doing better in school. 

• Parents and students also report improved study skills and increased motivation to leam 
as a result of participation in ESS. Students appreciate having opportunities to make up 
or retake tests. This flexibility for students who either missed or performed poorly on a 
test indicates that value is placed on allowing students the opportunity to show what they 
have learned. 

• For many students, ESS provides a time to receive individualized instruction, to leam 
study skills, and to have learning reinforced through the use of games, visual aids, 
practice, additional time, and incentives. 
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Program Strengths 

• Major strengths of the ESS program focus on processes for its implementation. For 
instance, process-linked supports include targeting students as early as possible, 
dedicated staff, student transportation, collaboration between teachers and coordinators, 
flexible scheduling, low teacher/student ratio, and individualized instruction. 

• There is a high degree of continuity between coordinators’ and teachers’ beliefs about 
key forces that help the ESS programs to succeed. The most critical components for 
successful implementation are strong district- and building-level support. Other critical 
components for implementation success are collaboration and relationships among staff, 
parent or community support, staff development, and financing. 

• Coordinators’ responses confirm that there are numerous successful programs operating 
in many schools in Kentucky. One particular reason given for success was the use of 
innovative and creative ESS methods. 

• One unique strength of the ESS program is its fluidity and flexibility. Student mobility is 
high throughout the program. As a particular problem arises, ESS allows for an 
immediate intervention that focuses on a specific need that can be addressed before it 
becomes chronic and long term. The program does not rely solely on the results of 
annual standardized test scores, which would slow down the process of identification, 
referral, and enrollment. 

Barriers to Maximum Success 

• A variety of topics are viewed both as weaknesses and as strengths, depending on their 
presence or absence. These include student transportation, funding, staff development, 
parental communication, staffing, and student motivation. This suggests that when these 
factors are in place and sufficient, they provide a strong foundation for successful ESS 
implementation. Conversely, the absence or insufficiency of these factors is detrimental 
to maximizing the potential of an ESS program. These issues are more fully discussed in 
the context of weaknesses so that administrators and policymakers can see the 
explanatory comments related to each. 

• Student transportation is a major problem for some schools. The decision to use ESS 
funds to provide public transportation for students is detennined by individual school 
and/or district policies. Because the majority of the ESS services offered during the 
regular school year occur after nonnal school hours, if bus service is not provided then 
parents must make transportation arrangements for their children. With the combination 
of parental work schedules, a potential lack of transportation for lower-income families, 
and the distance involved for more rural communities, this factor could seriously deter 
participation of some students who might be most in need of such academic services. 
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• Staff development related to ESS now seems to be nonexistent, inadequate, or distributed 
unevenly between teachers and coordinators. This may be more problematic for newer 
staff members who are initially becoming involved with ESS and who are not familiar 
with its related philosophies and guidelines, especially since the ESS summer conference 
was discontinued. Further, there is some lack of agreement among school coordinators, 
teachers, parents, and students as to the exact intent and nature of the ESS program. 

• One discrepancy noted among respondent groups involves communication, especially 
with parents. While ESS teachers believe they meet with parents on an as-needed basis, 
parents note that communication with the teacher about their children’s progress is a 
major problem and that they often are not aware of ESS goals. 

• There seems to be some degree of misunderstanding regarding the emphasis on core 
subjects taught in ESS sessions. District coordinators’ perceptions seem to be most 
closely aligned with the parameters of the ESS policy and regulations. 

• Most of the respondents believe that the current number of teachers involved in ESS is 
inadequate for the number of students. Related to this topic is the reported difficulty 
associated with recruiting, hiring, and retaining a sufficient number of interested teachers 
with appropriate content knowledge and relevant skills for working individually with 
students in the ESS environment. 

• Student motivation is a relevant issue for encouraging participation in the ESS program. 
Although some students are not motivated enough to participate, those who do participate 
tend to become more interested and to improve their academic perfonnance as a result. 
Moreover, students consider the use of alternative, “fun” instructional strategies in the 
ESS classrooms as more engaging. 

• Finally, there is consent among the coordinators and teachers that additional funding is 
necessary to adequately support full implementation of the ESS program. Addressing 
several of the weaknesses noted above would require an increased level of funding to 
provide consistent student transportation, staff development, expanded services in terms 
of hours and/or subjects, and a reduction of the student/teacher ratio. 

Program Fidelity 

• The ESS programs are performing satisfactorily in terms of implementing the majority of 

the 15 major components of the statewide program. The following four components 
seem to be implemented least satisfactorily: staff selection, instructional resources, 

collaborative planning processes, and program evaluation. 

• There are four types of implementation of ESS programs in tenns of their fidelity in 
operating the 15 major components of the program. That is, there are four levels of 
implementation of the ESS program, ranging from high-fidelity implemented to low- 
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fidelity implementers. However, these patterns of implementation are very similar across 
the four groups; the main differences are in the levels of implementation of each 
component, as opposed to the differences across the components. Three of the four high 
implementation schools are middle schools with small ESS programs in terms of the 
number of involved students and teachers. In other words, the high-fidelity 
implementation is more an artifact of program scale and building level rather than 
discrete differences in implementation. 

Patterns of Implementation 

• Although there seem to be no discernable operational differences in the four levels of 
implementation, there are some differences in associated measures when compared by 
implementation pattern. The high implementation group consistently spent less time on 
teacher-led activities and more time on student-led instructional activities than any of the 
remaining three groups. 

• When looking at implementation patterns with other data measures utilized in this 

comprehensive evaluation, one other conclusion can be drawn: All the ESS school 

coordinators in the high implementation group pinpointed inadequate financial support. 

Overall 

• One of the most striking conclusions from this comprehensive evaluation of the statewide 
Kentucky Extended School Services program is the marked consistency and high degree 
of corroboration both within and among respondent perceptions and data collector 
observations. 

• Overall, it is concluded that the ESS program is positively perceived by involved 
stakeholders and has been proven to help address the needs of students who are at risk 
academically. However, several areas have been identified in which improvements could 
be made for a more successful implementation of the statewide program. 
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RECOMMENDATIONS 



Based on the findings and conclusions of the statewide Kentucky Extended School 
Services evaluation, a number of specific recommendations were offered for KDE staffs review 
and reflection. 

• Some thought should be given to making scheduling adjustments to the after-school ESS 
programs, such as expanding hours of operation or simply staggering scheduled times 
within a week. The demand for additional time in ESS will need to be weighed against 
the possibility of further exacerbating the conflict between ESS sessions and 
extracurricular activities and/or part-time jobs. 

• ESS staff should encourage/facilitate more involvement of parents and students in setting 
goals for individual students. This would help to improve communication between the 
home and school and to ensure that all involved parties share similar goals for individual 
students’ learning — further increasing the likelihood that these goals will be uniformly 
sought, supported, and achieved. In addition, continued communication with parents 
about their children’s progress should be a routine part of ESS program operation. 

• Professional development opportunities should be provided to ESS coordinators and 
teaching staff in the areas of staff selection, instructional resources, collaborative 
planning processes, individualized instruction, mentoring/tutoring, and program 
evaluation. The specific format for these professional development opportunities could 
vary from workshop sessions at a central site or decentralized sites to online, Internet- 
based courses. Whatever delivery method is selected, professional development in these 
four areas is needed by most ESS program staff in the state. 

• School-level ESS staff should carefully consider the scale of the program as they plan, 
deliver, and evaluate their programs to improve the level of implementation. Rather than 
resorting to downsizing, ESS staff need to assess how thoroughly and effectively they 
have implemented the 15 major program components and develop an action plan for 
improving those areas identified as being low or poor. 

• KDE staff and state board of education members should collaborate to identify possible 
solutions to transportation issues. Solutions might include working closely with 
transportation staff, investigating alternative funding formulas such as using non-ESS 
monies for transportation expenses and/or seeking additional funds specifically for 
transportation. 

• Some thought should be given to exploring ways to overcome the teacher staffing issue. 
For example, KDE staff could identify those districts experiencing ESS teacher 
recruitment problems and work with them to develop solutions. If the problem is teacher 
pay for ESS sessions and state or local regulations that prevent increasing teacher 
salaries, perhaps KDE staff could be instrumental in finding ways to overcome those 
barriers, such as seeking waivers for current rules or regulations. 




The possibility of developing an incentive program for ESS teachers that would generate 
opportunities for recognition of their efforts should be investigated. For example, an ESS 
Teacher of the Year award program might be designed and implemented. The idea is to 
offer a significant award and possibly a financial reward based on state-established 
criteria. The award, which could be regional or statewide, may help draw teachers 
previously uninterested in participating in the ESS program. 

The summer conference for ESS coordinators and teachers should be re-instituted. This 
conference provides an excellent opportunity for numerous professional development 
sessions for ESS coordinators, teachers, and staff from any district. Also, the 
opportunities to share ESS program information, successes, and solutions to common 
problems would be greater at a large conference. The added value would be that 
professional associations and networking about ESS across the state would likely evolve 
from such a conference; for example, a statewide organization of ESS professionals. 

It should be clearly communicated to all stakeholders that the ESS program, as 
implemented under current laws and regulations, is not designed to be an enrichment 
program. A clear understanding of the specific nature and purpose of the statewide 
program may help avoid efforts to shift its focus from struggling learners to all students. 

The current mechanism of categorical funding for the individual ESS programs should be 
maintained. Nearly all district and school respondents agreed this system worked well 
and felt that funds were distributed equitably. 

KDE and local ESS school staff should investigate ways to recruit at-risk and hard-to- 
reach students. Identified successful methods could be included in the best practices 
resource described below. Schools or districts could apply for grant money to fund 
focused, intensive efforts to increase students' awareness of and interest in the ESS 
program. Other possibilities include modifying current ESS activities to make them more 
fun for students by introducing creative, innovative instructional strategies to better 
capture students' interest, or experimenting with an incentive system to provide more 
extrinsic, short-tenn rewards to give students a sense of accomplishment during their 
participation in the ESS program (in addition to the intrinsic, long-term goal of increasing 
their academic achievement). 

KDE staff should formalize and fund the process for obtaining ESS “best practices” and 
develop a resource tool that would be available to all ESS staff. ESS staff in one or more 
districts could be financially compensated for spearheading the initiative and gathering 
submissions from all ESS programs. The final product could be in print or electronic 
format and would be a compendium of innovative and creative ESS programs. It could 
also include a segment on student motivation, as mentioned earlier. We understand that 
such an effort is currently under way, but statewide coordinators indicated limited 
awareness of this undertaking. Therefore, at the very least, KDE staff should increase the 
visibility and potential utility of such a tool for the ESS program statewide. One potential 
resource is the Promising Practices in Afterschool (PPAS) Web site, which provides 
detailed descriptions of promising practices nationwide (see www.afterschool.org). 
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DISCUSSION 



This section presents a discussion of the results of the mixed-method evaluation of the 
Kentucky ESS program. The focus here is not as much on the findings themselves as on how the 
methods used contributed to discovering corroboration and contradiction in the findings. Using 
several types of data, usually from a combination of qualitative and quantitative data collection 
methods, has become very common in the field of evaluation (Datta, 1997; Lawrenz & Huffman, 
2002). Caracelli and Greene (1997) described two broad classes of design options that further 
clarify the myriad purposes for using mixed method designs: component and integrated designs. 
One component design, triangulation, formed the basis of the design of the current evaluation, 
which, although it included qualitative elements, leaned heavily toward the positivist, 
quantitative end of the spectrum. Triangulation entails assessing the convergence, congruence, 
corroboration, and/or correspondence of results across different methods of investigating the 
same phenomenon in order to increase validity (Caracelli & Greene, 1997; Greene, Caracelli, & 
Graham, 1989). We also used this multi-method design in order to look for any contradictions in 
findings. 



Congruence 

Referral to ESS/purpose of ESS, Students, teachers, coordinators, and parents who 
were surveyed were in agreement that students were referred to ESS so that they could improve 
their academic performance. This confirms that those at the school level understood the explicit 
purpose of ESS as set by the state legislature. Nearly all of the 151 district coordinators who 
responded to the district coordinator survey indicated that most students were referred to the 
program because they were in danger of failing and needed to improve their academic 
performance. 

The ESS teachers who completed an ESS teacher survey tended to corroborate this view 
of student perfonnance. Asked to select from a list the most common reasons students receive 
ESS services, 84% indicated that they were to improve their academic achievement, and three 
fourths noted that students were in danger of failing. Similarly, nearly three fourths of the non- 
ESS teachers responding to the survey reported that their students received ESS services because 
they were in danger of academic failure and 73% agreed that students were referred to ESS to 
improve their academic achievement. 

Interviews with parents of students participating in ESS confirmed this finding, given that 
22% of the parents who were interviewed indicated that their children were participating in the 
program because they needed to improve their grades in a specific subject. In addition, 19% of 
the ESS students interviewed indicated that they were participating in ESS because they needed 
to do better in school, had failed a class, needed to improve a specific grade, or needed to 
graduate. Thus, from all these sources it seems that student performance and success in school 
were clearly the primary reasons that students received ESS services. 
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The innovation configuration map scores for student eligibility lent further credence to 
the findings from other methods that the ESS programs were clear about student eligibility rules. 
The mean scores on the 1 to 3 rubric ranged from 2.21 on student assessment for eligibility to 
2.38 on entry and exit process to 2.46 on referral guidelines and student selection. 

Facilitators of success. There is a high degree of continuity between coordinators’ and 
teachers’ beliefs about key forces that help the ESS programs to succeed, as reported in the surveys. 
Between two thirds and three fourths of all coordinators (school- and district-level) and ESS teachers 
reported excellent staff relations were facilitating the success of ESS at their school. 

Student outcomes. There is congruence among perceptions of the intended and actual 
outcomes of the ESS program. All stakeholders agree that the ESS program is helping students 
increase their academic achievement, pass courses and grades, and decrease school failure. 
Eighty-nine percent of non-ESS teachers, 95% of ESS teachers, 98% of district coordinators, and 
99% of school coordinators agreed that the program enhanced student academic performance. 
The majority of parents (78%) and students (78%) also reported that students were doing better 
in school after they started attending ESS. 



Contradictions in Findings 

Source of referral. There is a great deal of consistency among the perceptions of 
coordinators, teachers, and parents as to how students are referred to ESS; the majority believes 
that students are referred most often by classroom teachers. However, 80% of interviewed 
students reported that they most often self-select into the program. It may be that students are 
taking credit for self-selection by agreeing to participate in this voluntary program after a teacher 
or parent has made the suggestion. Either approach seems to allow enough flexibility for the 
intended population to become involved with the program. 

Reasons for referral. As stated under the Congruence subsection, there was much 
agreement across methods and role groups as to the purpose of the ESS program. However, 
there was one small area of contradiction regarding the major reasons for referral to ESS. 
According to 95% of the 151 district coordinators who replied to the survey, most students were 
referred to the program because they were in danger of failing. On the other hand, of the 837 
school ESS coordinators who returned their surveys, only 76% indicated that students were 
referred to the program because they were in danger of failing. This represents a difference of 19 
percentage points between the reports of district and school coordinators. 

Data gathered during site visits suggested that students referred to ESS were at some risk for 
failure, but not to the degree suggested by respondents to the district and school coordinator surveys. 
Of the 15 district coordinators interviewed during AEL site visits, for example, fewer than half (45%) 
reported that students in their districts were recommended for program participation because they 
were not succeeding academically. Similarly, only 43% of the 23 school coordinators interviewed 
thought that students were referred to ESS because their perfonnance was deficient. Differences in 
responses may be due to the nature of the methodology — the surveys forced them to choose among 
pre-listed response options whereas the interviews were open-ended. 
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Perceptions of outcomes. There was a difference in the perceptions of student outcomes 
between those teachers who were not involved with the program (non-ESS teachers), and the 
staff groups who were involved with the program (ESS teachers, school coordinators, and district 
coordinators). Those involved with ESS (teachers and coordinators) were one and one-half to 
two times more likely as non-ESS teachers to report increases in student self-esteem or 
motivation as outcomes of the program. They were also two to three times more likely than non- 
ESS teachers to report improved attendance as an outcome. 

Home/school communication. One discrepancy noted among respondent groups 
involves communication, especially with parents. While ESS teachers believe they meet with 
parents on an as-needed basis, parents note that communication with the teacher about their 
children’s progress is a major problem and that they often are not aware of ESS goals. Parents 
and teachers also had different perceptions of the quality of communication. Teachers generally 
reported having had more contact with parents than parents reported having with teachers. The 
same phenomenon occurred with the teachers and students — teachers reported better 
communication with students than students reported having with teachers. This could be a 
“perspective” issue. Students and parents have only their own communication with the ESS 
teacher to consider, whereas a teacher may think communication is more frequent with parents 
and students because he or she is generalizing across all the students under his or her care. 



Summary of Discussion 

The comprehensiveness of this evaluation permitted many findings to be corroborated or 
expanded upon using multiple methods. The strength of the methodology employed is evidenced 
by the richness of the data, which is only touched upon in this paper. Having findings that are 
confirmed through multiple perspectives and multiple methods allows greater confidence in the 
findings. The statewide surveys and the surveys administered during the site visits provided 
valuable data on the ESS program. However, had the evaluation design been limited to these 
data, the evaluators would not have been able to form most of the conclusions or formulate many 
of the recommendations that were included in the final report. The interview data elaborated on 
the survey findings, as did the observational data and the innovation component configuration 
map; yet they, too, would have been insufficient on their own. The triangulation identified a 
number of contradictions across data sources. While some of the discrepancies may be 
insignificant, identification of such contradictions may bring to light a critical factor of program 
success or failure across sites. 

The methodology employed in this effort seemed to serve well the intended purposes of 
the evaluation. We were able to gather a great deal of data from a variety of sources and in a 
variety of ways. This was critically important considering the ultimate audience of the 
evaluation was the state legislature. The need to satisfy the research “preferences” of diverse 
policy makers, including those who are satisfied only by rigorous quantitative designs and those 
who want to develop a better understanding of the different ways the program personally affects 
participants, was central to this evaluation. The methodology allowed the evaluators to find 
numerous areas of congruity and relatively few areas of contradiction or discrepancy. 
Furthermore, the ability to achieve elaboration was invaluable in helping the evaluators to gain a 
deeper understanding of just how the ESS programs were having an effect. 
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